Interaction Sensing

ABSTRACT

In particular embodiments, an apparatus includes an insulator coupled to one or more electrodes that are configured to passively sense charge displacement or a change in characteristics of electromagnetic signals in an environment.

PRIORITY

This application claims the benefit, under 35 U.S.C. §119(e), of thefollowing, which are all incorporated herein by reference: U.S.Provisional Patent Application No. 61/865,448, filed 13 Aug. 2013; U.S.Provisional Patent Application No. 61/924,558, filed 7 Jan. 2014; U.S.Provisional Patent Application No. 61/924,604, filed 7 Jan. 2014; U.S.Provisional Patent Application No. 61/924,625, filed 7 Jan. 2014; U.S.Provisional Patent Application No. 61/924,637, filed 7 Jan. 2014; U.S.Provisional Patent Application No. 61/969,544, filed 24 Mar. 2014; U.S.Provisional Patent Application No. 61/969,558, filed 24 Mar. 2014; U.S.Provisional Patent Application No. 61/969,590, filed 24 Mar. 2014; U.S.Provisional Patent Application No. 61/969,612, filed 24 Mar. 2014; andU.S. Provisional Patent Application No. 62/000,429, filed 19 May 2014.

TECHNICAL FIELD

This disclosure generally relates to electronic devices that detectinteractions with objects, and more particularly to devices that usesurface contact sensors or proximity sensors to detect interactions.

BACKGROUND

A touch sensor can detect the presence and location of a touch or objector the proximity of an object (such as a user's finger or a stylus)within a touch-sensitive area of the touch sensor overlaid on a displayscreen, for example. In a touch sensitive display application, the touchsensor may enable a user to interact directly with what is displayed onthe screen, rather than indirectly with a mouse or touch pad. A touchsensor may be attached to or provided as part of a desktop computer,laptop computer, tablet computer, personal digital assistant (PDA),smartphone, satellite navigation device, portable media player, portablegame console, kiosk computer, point-of-sale device, or other suitabledevice. A control panel on a household or other appliance may include atouch sensor.

There are a number of different types of touch sensors, such as, forexample, resistive touch screens, surface acoustic wave touch screens,and capacitive touch screens. Herein, reference to a touch sensor mayencompass a touch screen, and vice versa, where appropriate. When anobject touches or comes within proximity of the surface of thecapacitive touch screen, a change in capacitance may occur within thetouch screen at the location of the touch or proximity. A touch-sensorcontroller may process the change in capacitance to determine itsposition on the touch screen.

Capacitive touch operates by sending a signal from an electrode, andthen measuring the variation caused by the presence of interveningmaterials. Actively emitting an electric field adds to the energy usageof the device and slows down responsiveness. Additionally, scaling thecapacitive touch sensor to very large areas can be cost-prohibitive.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate an example TriboTouch system, which candetermine positions of an object based on triboactivity.

FIGS. 2A-2E illustrate an example interaction between a finger and aTriboTouch sensor.

FIG. 3 illustrates an example architecture of a TriboTouch system.

FIG. 4 illustrates an example alternative analog front-end.

FIG. 5 illustrates principles of TriboTouch operation.

FIG. 6 illustrates an example process for determining types of contactsbased on signal profiles.

FIG. 7 illustrates an example of combining the capabilities ofcapacitive sensing and TriboTouch.

FIG. 8 illustrates an example of capacitively coupling a transmitter toan electrode while using the same receiver system for both capacitiveand TriboTouch sensing.

FIG. 9 illustrates a triboactive surface covered with an array ofdifferent materials.

FIGS. 10A-10C illustrate different positive and negative charge patternsgenerated when different objects make contact with the same patternedarray of sensors.

FIG. 11 illustrates an example configuration of a NoiseTouch systemrelative to a user and the environment.

FIG. 12 illustrates an example NoiseTouch system architecture.

FIG. 13 illustrates an example process that determines hand poses orpositions.

FIG. 14 illustrates an example method of separating touch and stylusdata.

FIG. 15 illustrates detection of the signal modification characterizingthe modification of ambient noise by the contact by a stylus or pen.

FIG. 16 illustrates an example process of passively sensing theenvironment and context of a user.

FIG. 17 illustrates examples of noise contexts that can be passivelysensed.

FIG. 18 illustrates an example process of using the context sensingsystem to communicate with a device having a NoiseTouch sensor.

FIG. 19 illustrates an example architecture of a TriboNoiseTouch system.

FIG. 20 illustrates an example method of separating triboactive datafrom noise data.

FIGS. 21-23 illustrate example TriboNoiseTouch processes for identifyingtriboelectricity-related events and noise-related events.

FIG. 24 illustrates a triboactive subsystem producing examplehigh-resolution data based on individual micro-contacts with a surfaceof a touch sensor, while a noise-based sensing subsystem produces anexample blob around the area of contact or hover as well as a “shadow”of a hand hovering over the surface.

FIG. 25 illustrates an example method enhancing the accuracy of fingercontact.

FIG. 26 illustrates an example method for detecting a finger contact andisolating it from a non-conductive pen contact.

FIG. 27 illustrates example estimation of a pen or hand pose bydetecting a hover shadow of the hand making contact or holding the pen.

FIG. 28 illustrates example TriboTouch sensing for providing highresolution stylus sensing and example TriboNoise sensing for detecting aspecifically designed stylus that features buttons to trigger menus andfunctions.

FIG. 29 illustrates an example method for improving a dynamic range forhover sensing.

FIG. 30 illustrates example single-touch electrode components.

FIG. 31 illustrates two electrodes in an example interleaved pattern.

FIG. 32 illustrates a row-column electrode grid that can be used todetect position of two touch points.

FIGS. 33 and 34 illustrate array multitouch configurations usingsingle-touch electrodes in a grid.

FIG. 35 illustrates an example of continuous passive position sensingusing a resistive sheet electrode.

FIGS. 36 and 37 illustrate an example of continuous two-dimensionalpassive position sensing.

FIGS. 38-40 illustrate example electrode-sheet configurations.

FIG. 41 illustrates an example of dielectric-encoded passive positionsensing.

FIGS. 42 and 43 illustrate an example of continuous passive positionsensing using an array of non-linear elements.

FIG. 44 illustrates an example of spatially-distributed coordinatedencoding.

FIG. 45 illustrates an example combination of TriboTouch with resistivetouch sensors.

FIGS. 46 and 47 illustrate example combination of TriboTouch withinductive touch sensors.

FIG. 48 illustrates an example computer system 4300.

DESCRIPTION OF EXAMPLE EMBODIMENTS

FIGS. 1A and 1B illustrate an example TriboTouch system, which candetermine positions of an object based on triboactivity. FIG. 1A showsan insulator surface adjacent to an electrode. The electrode isconnected to TriboTouch hardware, which determines the positions of anobject 130 such as a finger when contact of the object with theinsulator produces a local charge displacement, as shown in FIG. 1B. Thecharge displacement is not a net current flow, but rather a chargedisplacement that is reversed when contact with the object is removed.This distortion of the internal electric field of the insulator can bepicked up by the TriboTouch hardware, and interpreted as contact andseparation events. Additionally, the distortion spreads out over aregion from the point of contact, allowing for a continuous estimate ofposition.

FIGS. 2A-2E illustrate an example interaction between a finger and aTriboTouch sensor, which can be used to determine the finger's positionbased on triboactivity. When two objects come into contact, charge canbe transferred between them due to the interaction of electron cloudsaround the surface atoms. This effect is known by various names,including triboelectricity, contact potential difference, and workfunction. In the semiconductor industry these phenomena lead toelectrostatic discharge (ESD) events which can damage sensitiveelectronic devices. Rather than attempting to mitigate these effects,techniques disclosed herein referred to by the name “TriboTouch” usethis charging mechanism to detect surface contact and motion effects.TriboTouch can directly sense insulators (e.g., gloves, brushes, etc.)as well as conductors or strong dielectrics (e.g., fingers, conductiverubber, etc.) to enable modes of interaction with sensing surfaces suchas those described herein. In one aspect, TriboTsouch uses local chargetransfer caused by contact and need not emit an electric field to bemeasured.

Aspects of a TriboTouch system are illustrated in FIGS. 2A-2E using afinger, but any conductive or non-conductive object can have the sameeffect. In one aspect, TriboTouch works by measuring the chargedisplaced when two objects come into contact or separate. No secondarymechanism is needed to induce charge in the object being sensed. Thereis no need to transmit a signal to be measured. Instead, charge isgenerated and received when an object contacts the sensing surface. FIG.2A shows a finger above an insulating surface. In FIG. 2B, as the fingercontacts the surface, charge flows and current is sensed. In FIG. 2C,current ceases at equilibrium. In FIG. 2D, finger separation causescharge redistribution and an opposite current. In FIG. 2E, equilibriumhas been restored.

Charge transfer can occur between combinations of insulators,semi-conductors, and conductors with dissimilar surface properties(e.g., composition, surface microstructure, etc.). The polarity, surfacecharge density, and rate of charge transfer (“contact current”) dependon the particular materials involved. The amount of charge transferredbetween two materials can be estimated from their relative positions inan empirically-determined “triboelectric series”. A commonly-acceptedseries, ordered from most positive to most negative, is: air, human skinor leather, glass, human hair, nylon, wool, cat fur, silk, aluminum,paper, cotton, steel, wood, acrylic, polystyrene, rubber, nickel orcopper, silver, acetate or rayon, Styrofoam, polyurethane, polyethylene,polypropylene, vinyl (PVC), silicon, and Teflon (PTFE). TriboTouchallows detection of contact by essentially any solid material.

FIG. 3 illustrates an example architecture of a TriboTouch system. Ahigh impedance amplifier 306 amplifies incoming signals 305 receivedfrom an input electrode 304 in response to a surface contact 302, and asubsequent analog to digital converter (ADC) converts this signal 305 todigital form. The input electrode 304, high impedance amplifier 306, andADC 308 convert the signal 305 as seen at the electrode 304 accuratelyto digital form. Other embodiments can use sigma-delta approaches,charge counting, charge balancing, or other means of measuring smallcharge displacements, as shown in FIG. 4. A gain control system 310 canoptionally be used to maintain the values within the prescribed range ofthe system. In one or more embodiments, the components that receive andconvert the input signal to digital form are referred to herein as ananalog front-end. The analog front-end can include the input electrode304, amplifier 306, ADC 308, and gain control 310, or a subset of thosecomponents. A processing system 312 receives the digital signal andgenerates position data 332. The processing system 312 can beimplemented using hardware, software, or a combination of hardware andsoftware. The processing system 312 starts at block 314 and performsinitial calibration at block 316. Then the baseline can be determined byan adaptive method 318. The adaptive method can be, for example, arunning average, a differential measurement with respect to a shieldelectrode, or a composite measure computed from an aggregate ofmeasurement sites, or other methods. This may be triggered as the systemis first initialized, or when the system detects that there is a driftin the signal, as indicated by a constant offset of values over a longperiod. Once this baseline is subtracted at block 320, the noise in thesignal (for example to detect common 50/60 Hz noise, and frequenciesabove and below the expected range of the system) is modeled andrejected at block 322, leaving a signal due to contact charging effects.Contact charging events are then detected and classified, at block 326,as contact, separation, or motion by their time domain profile, usingmethods such as matched filters, wavelet transforms, or time-domainclassifiers (e.g. support vector machines). These events are thenintegrated by a state machine at block 328 to create a map of contactstates on the sensing surface, which allows the system to track when andwhere contact and release events take place. Finally, this map is usedto estimate event types and coordinates at block 330. Note thatTriboTouch does not ordinarily produce a continuous signal when acontact is stationary. However, it does produce opposite-polaritysignals on contact and removal. These opposite-polarity signals can beused to keep track of how additional contacts are formed and removed inthe vicinity of an existing contact point. The pattern of contacts canbe understood by an analogy to the effects of dragging a finger throughsand, in which a wake is formed before and after the finger. Similarly a“charge wake” is seen by the system, and the charge wake is used todetermine motion. The final output is a high-level event stream 333describing the user's actions. The output can include position data 332.In one or more embodiments, large objects, e.g., a finger, can bedistinguished from a collection of multiple touch sites because largeobjects tend to produce a larger “imprint” of contact that is receivedat substantially the same time. By correlating the contacts in time, theTriboTouch system can keep track of which contacts belong together. Evenwhen two objects are in close proximity, as in a pinch gesture, forexample, the sensor actually detects two contact “peaks” very closetogether. Therefore the contact relationships can be maintained.

FIG. 4 illustrates an example alternative analog front-end. While thedescription of FIG. 3 has been related to the use of a high-impedanceamplifier 306 followed by an analog-to-digital converter 308, TriboTouchcan also employ a charge-balancing sigma-delta converter, or it cancombine both approaches. In the configuration shown in FIG. 4, acapacitor 406 is switched by a switch 404 between a reference voltagesource (Vref) 408 and the input electrode 402 to transfer packets ofcharge, thereby keeping the input electrode potential within the rangeof the input amplifier 410 (or comparator in the case of a 1-bitsigma-delta ADC). The subsequent signal processing chain combines theoutput 315 of the ADC 412 and output of the automatic gain control (AGC)414 to reconstruct the input current with a higher dynamic range thanwould be possible with the input amplifier and ADC alone. Thereconstructed input current is provide to TriboTouch signal processing416, which can be the processing system, 312 or other signal processingsystem.

As described above, TriboTouch can sense signals directly generated byphysical contact and need not transmit signals to be sensed. Therefore,the system does not emit spurious signals as a result of its activitiesoutside of what may be normally expected from any electronic circuit,simplifying compliance with EMI regulations and design ofnoise-sensitive electronics positioned nearby. An additional benefit isthe power savings from this design. There is direct savings from nothaving to transmit a field. Additionally, the system benefits from asimplified architecture, which means there are fewer electronic devicesto power. Further, since there is no need to perform extensive noiserejection in hardware, there can be additional savings from reduction ofcomplexity.

FIG. 5 illustrates principles of TriboTouch operation. The tribochargingcaused by contact with the insulating surface is coupled capacitively tothe electrode via dielectric polarization. TriboTouch is thus capable ofdetecting contact, motion, and separation of objects at the surface ofthe insulator. As such, it is possible to use any object (finger, glove,plastic stylus, paint brush, paper, etc.) to interact with the sensingsurface. A data processing system can determine the type of object thatinteracts with the surface using an event detection and classificationcomponent 506. The event detection and classification component 506 usesclassification characteristics 504 to determine contact type data 508,which identifies the type of the object. The classificationcharacteristics 504 can include one or more signal patterns 502 thatcorrespond to different types of objects. For example, a first signalpattern 512 can correspond to a finger, a second signal pattern 514 to aglove, a third signal pattern 516 to a plastic stylus, a fourth signalpattern 518 to a paint brush, and so on. The event detection andclassification component 506 can, for example, compare the detectedtribocharging signal to the signal patterns 502 and select one of thesignal patterns 502 that best matches the detected signal. The eventdetection and classification component 506 can also estimate theposition 510 of the detected signal, as described above with referenceto FIG. 3.

FIG. 6 illustrates an example process for determining types of contactsbased on signal profiles. Because TriboTouch can sense contact with,motion across, and separation of an object from the sensing surface, itis not necessary to algorithmically derive these events from capacitancemeasurements. TriboTouch can therefore produce more accurateidentification of these events than capacitive sensing can ordinarilyprovide. Additionally, because of the localized nature of tribocharging,the position estimation algorithms can yield higher spatial and temporalresolution than capacitive sensing methods. This higher resolution canbe used, for example, to perform palm rejection or other inadvertentcontact rejection using the process shown in FIG. 6. The process of FIG.6 detects and classifies events at block 602. Block 604 integratesevents, e.g., by using a state machine to create a map of contact stateson the sensing surface, as described above with reference to FIG. 3. Themap can be used to track when and where contact and release events takeplace, and to estimate event type and coordinates. Block 608 estimatesevent positions to generate position data 612. Block 610 detects posesto generate hand and stylus pose data 614. As shown in FIG. 5, differenttypes of contacts can have different characteristic signal profiles(examples shown are not indicative of actual material data) and thereceived signal's characteristics can be used to detect, for example,inadvertent palm contact while sketching with a stylus. In one or moreembodiments, different object types can be detected based on the contactprofiles of the objects without a special pickup design. These profilescan be expressed either in terms of example waveforms, or in terms ofalgorithmic approaches that capture distinctive features of thewaveform.

The TriboTouch system can use one instance of the above-mentionedhardware for each touch position, or it can use a continuous largerelectrode, and estimate the position based on the distance-dependentchange in signal through the electrode. The change can be caused bymaterial properties of the covering material, resistance of theelectrode body, reactive impedance of the electrode, or any othermethod. TriboTouch can therefore distinguish position at a resolutionhigher than that of its electrode structure. In one or more embodiments,when an instance of the hardware is used for each touch position, thehardware instances operate in parallel, so that each electrode ishandled individually. The parallel arrangement allows faster readspeeds, but increases hardware complexity. Alternatively, scanningthrough each electrode in sequence offers different tradeoffs, becausethe digitization system should be faster (and thus consume more power),but the overall system is more compact (which can reduce the powerconsumption).

TriboTouch can be configured for single or multiple touch points, andadditionally can be configured for either continuous position sensing(such as a phone or tablet), or discrete position sensing (such as abutton). That is, position and motion can be sensed, as in atouchscreen, or discrete switches can be used. In one example, a4-contact resistive-pickup system can be used. Alternatively, arow-column system that detects 2 simultaneous contacts can be used. Asanother alternative, pickups can be added to a resistive system. Inanother example, an array of pickups can be used to detect 5 contacts.The specific pickup configuration is a design option for the pickups andelectronics. In the discrete position sensing applications, thestrengths of the system remain in force, and make the system practicalto use in many scenarios where environmental noise or contamination maybe an issue, such as in automotive or marine uses, in factory floors,etc. In such cases, TriboTouch can provide the benefit of robust inputwithout the need for additional precautions necessary for traditionalcapacitive sensing.

FIG. 7 illustrates an example of combining the capabilities ofcapacitive sensing and TriboTouch (e.g. direct sensing of contact withconductive and non-conductive objects). Because both strategies usesensitive measurements of charge displacement, it is possible to combinethem using essentially the same analog front end hardware. FIG. 7illustrates the basic principle whereby an electrode 702 can be sharedbetween these two sensing methods. Capacitive sensing works bytransmitting a balanced AC signal at high frequencies (typically >125kHz) using a transmitter 706 into an electrode, and measuring either thetransmit loading or the signal received at other electrodes. Thecapacitive measurement can be performed by a capacitive receiver 708.TriboTouch, however, works by measuring the local charge displacementusing a receiver 712 at low frequencies (ordinarily <1 kHz). Bycapacitively decoupling the electrode 702 from the capacitive sensingcircuit 708 using a capacitor 704, the triboelectric charge displacementcan be maintained and measured separately, either by time-multiplexingthe two sensing modes or by filtering out the transmit signal at theTriboTouch analog front-end or in subsequent signal processing. Whentime-multiplexing, the capacitive system 710 suspends access to theelectrode 702 while the TriboTouch system 714 measures, and vice versa.When filtering, the TriboTouch system 714 uses a filter and knowledge ofthe signal being sent by the capacitive system 710 to remove the effectsof capacitive measurements during the noise rejection phase ofprocessing. Further examples of combining other types of touch sensors,such as resistive, capacitive, and inductive sensors, are describedbelow with reference to FIGS. 45-47.

FIG. 8 illustrates an example of capacitively coupling a transmitter 804to an electrode 802 while using the same receiver system 806 for bothcapacitive and TriboTouch sensing. The capacitive software and theTriboTouch software can be combined into a single system 808. In thiscase, the capacitive software uses the same hardware as the TriboTouchsoftware, taking turns to use the shared resources.

FIG. 9 illustrates a triboactive surface covered with an array 900 ofdifferent materials. The embodiment shown in FIG. 9 makes it possible todistinguish between different contacting materials (e.g. skin, graphite,rubber, nylon, etc.) by patterning materials with differenttribonegativity over sensing sites 902, 904, 906 on the TriboTouchsurface. The principle is similar to that of a color CMOS image sensor,in which a color filter mask is superimposed over pixel sensors. In FIG.9, the triboactive surface is covered with an array 900 of fourdifferent materials, ranging from strongly tribopositive (++) materials902 to strongly tribonegative (−−) materials 906. When an objectinteracts with a cluster of these sensors, a characteristic chargepattern is generated that allows determination of the object's materialproperties (i.e. tribospectroscopy). In one or more embodiments, thearray can be laid over different electrodes. These electrodes can beclustered closely together, such that a small motion is sufficient tocross multiple electrodes. Differentiation between different materialtypes can be performed fewer material types to speed up type detection.

FIGS. 10A-10C illustrate different positive and negative charge patternsgenerated when different objects make contact with the same patternedarray 1008 of sensors. In FIG. 10A, contact with the finger 1002generates negative charge patterns on the −−, +, and − sensors, and aneutral charge pattern on the ++ sensor. Therefore, the finger 1002 ischaracterized by an overall strongly positive charge pattern. In FIG.10B, contact with the pencil 1004 generates positive charge patterns onthe + and ++ sensors, and negative charge patterns on the − and −−sensors. Therefore, the pencil 1004 is characterized by an overallneutral charge pattern. In FIG. 10C, contact with the eraser 1006generates positive charge patterns on the +, −, and ++ sensors, and aneutral charge pattern on the −− sensor. The eraser 1006 is thereforecharacterized by a strongly positive charge pattern. Thesecharacteristic charge patterns can be used to identify an unknown objectthat makes contact with the sensor array 1008.

TriboTouch allows for detecting a single contact, dual touch (e.g.,detect two fingers simultaneously making contact), multi-touch (e.g.,detect three or more fingers simultaneously making contact), the orderof touch (e.g., detect the order where index finger makes contact firstand then middle finger), the state of the object/finger where the firstobject/finger is in a first state and the second object/finger is in asecond state (for example, when rotating, the first finger can bestationary while the second finger rotates about the first finger),detect adjacent fingers versus non-adjacent fingers, detect thumb versusfingers, and detect input from prosthetic devices. TriboTouch alsoallows for detecting motion, and also detecting the position of thetouch/motion.

When contact is detected, TriboTouch allows for determining the shape ofthe object making the contact, the type of materials of the objectmaking the contact, activating controls based on the type of materialsthat are detected, activating modalities based on the shape and type ofmaterials detected (e.g., brush vs. eraser), using contact shape todepict contact realistically, using contact shape to detect object tochange modalities of application, and using contact shape to improveposition accuracy.

The dual touch detection allows for detecting zoom gesture, panninggesture, and rhythmic gesture to create shortcuts or codes. In addition,multi-touch detection allows panning gestures to control applicationswitching or multi-finger controls for games.

TriboTouch also allows the order of the touch to be detected so that,for example, rhythmic input can be used to create shortcuts or codes.Detecting adjacent fingers versus non-adjacent fingers can be used todetect input from chorded keyboard where multiple keys together form aletter. Detecting thumb versus fingers can be used to provide modifiedkeyboard input mode, allow for chorded input, and allow imprint offingers to be used as code. In addition, motion can be detected so that,for example, the following gestures can be detected: zoom in, zoom out,panning, dragging, scrolling, swipe, flick, slide, rotate clockwise, orrotate counterclockwise. The different types of contact,motion/gestures, and position described above can also be detected usingNoiseTouch and TriboNoiseTouch.

In industrial settings, the noise-resistance and distinctive signalcharacteristics of TriboTouch (and NoiseTouch) allow operation is noisy,humid, or dirty environments. These conditions generally prohibit theuse of capacitive sensors, and as a result the systems currently usedare relatively primitive (though robust)—such as physical buttons,membrane switches, IR touchscreens, etc. TriboTouch techniques enablethe same type of interfaces available to consumer users to be used inindustrial settings, such as to easy-to-clean hard glass touch controls,and so on.

TriboTouch can be used to provide self-powered buttons, e.g., fortransitioning from sleep mode to wakeup mode without capacitive sensing.When a contact takes place on a triboelectric control pickup, a smallcharge redistribution is triggered. As long as the electronics connectedto the device are sufficiently low-power, this displacement current maybe used to directly send a short message regarding the event.Alternatively, the device may collect power from static electricityproduced during bulk motion, and later use that power to operate duringrelevant contact events. This may be coupled with a radio transmitter orsimilar device to allow for completely wireless and battery-less remotecontrol of devices.

TriboTouch can provide indirect touch features, which can, for example,enable placing a paper on top of a touch screen and writing on the paperwith a finger, stylus, brush, or the like. TriboTouch (and NoiseTouch)surfaces operate with an insulator between the electrode and thecontacting object. However, the charge displacement effect can occur onany material. Therefore, the touch surface can be covered by anadditional material such as a sheet of paper or cloth, and operationwill not necessarily be impeded as a result. Since the contact of anytwo materials can produce a triboelectric effect, the makeup of the twomaterials making contact (while in contact with the touch surface),whether paper and pencil or brush and canvas, is not at an issue incontact detection.

Triboactive contact detection can be used to detect erasure, forexample, by detecting the motion of an eraser on top of paper, thusmirroring the digital content to what is drawn on the paper itself.Attachments can also be made to the screen to speed up particular input.For example, for gaming applications, a soft passive joystick that makescontact with the screen when pressed in different directions can be usedto provide the user with better haptic feedback. Similarly, a keyboardtemplate may be used to provide physical passive keys that can be usedto quickly trigger actions in applications such as drawings or 3Dgraphics, where switching between different modes and tools in rapidsuccession is common. Because triboactive contact sensing can sensecontact from non-conductive materials, the choice of materials for theattachments is greatly increased, and conductive or electrically activecomponents are not required. This allows a much broader class of inputexperiences at much lower cost, and use of a greater set of materialssuch as plastic, paper or wood for the attachments.

Gesture input can be provided using TriboTouch techniques. As discussedelsewhere herein, triboelectric charge displacement is analogous to thedisplacement of sand as a finger is run through it. A change in theangle of the finger (whether leaning left or right, angle of lean, andso on) can affect the way the sand is disturbed. Likewise, a change inthe angle of the finger can affect the charge displacement. This changein displacement can be measured to estimate the pose of the hand,including angle, handedness, and the like.

FIG. 11 illustrates an example configuration of a NoiseTouch systemrelative to a user and the environment. A person can be surrounded by anelectric field emitted by devices in the environment. This field isnormally considered part of the electromagnetic interference (EMI) inthe environment. This field is carried throughout the body, and can becoupled capacitively to electrodes in the device. Rather than attemptingto reject this noise, a technique referred to herein as “NoiseTouch”uses the noise that is conducted by the body and picked up by theelectrodes of the touch sensor to detect the position of the user'stouch. Feature parity with capacitive sensors is maintained (hoveringsupport, multitouch, etc.). NoiseTouch uses environmental noise and isthus immune to EMI, and does not need to emit an electric field to senseuser interactions. NoiseTouch is scalable, i.e., can be applied tosurfaces of any shape and size, responsive, and has reduced complexity.

Referring to FIG. 11, environmental EMI sources 1106 can be coupled tothe ground 1102 via impedance Z_(in) 1104, and to the human body 1110via impedance Z_(air) 1108. The body 1110 is also connected to ground1102 via an impedance Z_(b) 1112. The EMI 1106 is coupled to theelectrode 1118 through an optional insulator 1116, and is then receivedby the NoiseTouch hardware 1120, which itself is coupled to ground 1102via impedance Z_(h) 1114. The differences between impedance values toground of the different components in the system, and their exposure toEMI-induced electric field changes, result in a small potentialdifference sensed by the hardware 1120 from any source in the vicinity.In other words, when a large antenna, such as the human body 1110, is inproximity to the electrode 1118, the characteristic of the noise isdifferent compared to when the human body 1110 is not in proximity. TheNoiseTouch system 1120 can detect touch by sensing this change in thecharacteristics of the noise received by the electrode 1118.

FIG. 12 illustrates an example NoiseTouch system architecture.Environmental noise (from power lines, appliances, mobile and computingdevices, etc.) continuously emits electric fields that contribute to theenvironmental electromagnetic interference (EMI, i.e., electronicnoise). The human body is a slight conductor, and thus acts as anantenna for these signals. When the body closely approaches theelectrode, e.g., when the body is hovering over or touching a touchpanel, this signal is capacitively coupled to an input electrode 1206. Ahigh impedance amplifier 1208 amplifies the incoming signals, and asubsequent analog to digital converter (ADC) 1214 converts this signalto digital form.

In one or more embodiments, the processing system 1216 (e.g., processingsoftware running on a computer system) has two functions. Initially, theprocessing system 1216 characterizes the noise at block 1220 and adaptsthe gain at block 1218 so that the signal does not overwhelm theamplifier 1208. The data processing system 1224 then continues gainadaptation at block 1226, while rejecting unwanted signals at block 1228and estimating positions at block 1230. The gain adaptation informationis fed back to a gain controller 1210, which can be a portion of thefront-end hardware, to control the high-impedance amplifier 12078. Thegain adaptation maintains the signal from the amplifier 1208 within therange of the ADC 1214.

The noise characterization system 1220 can be used to break the noisesignal into bands and characterize the reliability of those bands basedon how constantly they are available, and what variability they exhibit.Via this analysis, a profile of each band is created, which can then beused by the noise source selection system 1222 to select an appropriateband (or set of bands) for position estimation. The selection processcan also decide to change the selection on a time-varying basis and theuser location and the noise environment around the user changes. Forexample, when the user sits down in front of a TV, a particular band maybe particularly fruitful. When leaving the home, this band may no longerbe as useful as that band (or set of bands) that are produced by thecar.

During operation, gain adaptation as described previously continues tooccur as necessary to keep the signal within range of the hardware.Using the characterization data, block 1228 removes the unwanted bandsof noise, and feeds the data to block 1230, which uses the signals toestimate where and how the user is approaching the surface. Block 1230also carries our linearization, such that the position of the user isexpressed as uniform values with relation to the edges of the surface.When used with an array of pickups, linearization in TriboTouch isessentially de-noising of the position data generated by the array.Because the positions are detected at each sensor, the position data iscleaned up and fit to a smoother motion. When used with the electrodepickup systems described herein (see, e.g., FIGS. 32-47, thelinearization system mathematically maps the positions from thecontinuous range of values produced by the system to the Cartesiancoordinates of the touch surface. In one or more embodiments, theprocess can be based on an individual mapping of touch positions toCartesian coordinates.

In FIG. 12, noise from the environment can be picked up by the user'sbody. This noise is capacitively coupled by the electric field of thebody to the input electrode 1206. The signal from the electrode is thendigitized. The digitization can be done by a number of methods,including the high-gain amplifier 1208 followed by the ADC 1214, asshown in FIG. 12. The signal can also be converted by other techniques,such as instrumentation amplifiers, sigma-delta converters, chargecounters, current metering approaches, and so on. The gain of the highimpedance amplifier 1208 can be optionally controlled by the gainadaptation component 1218 of the processing system 1216, though the highimpedance amplifier 1208 and ADC 1214 can alternatively have sufficientresolution such that gain control is not necessary. Followingdigitization, the data is fed to the processing system 1216, which canbe implemented in hardware or software. An initial calibration isperformed to set the gain if needed. Block 1220 characterizes the noiseinto frequency bands. The system can also determine the periodicity ofvarious noise bands. This determination allows for the selection of areliable band (or bands) at block 1222, based on continuous availabilityand signal strength. That information can be used to reject unwantedsignals at block 1228. Block 1230 performs position estimation. Duringthe course of processing, the signal characteristics may change. In caseof such a change, the system can trigger additional gain adaptation 1226or noise characterization 1220 to provide uninterrupted operations.Depending on the electrode structure, linearization can be performed atblock 1230 to compensate for the nonlinear nature of a continuous sheetelectrode, or a position can be estimated directly from the centroid ofthe activation seen in a row-column or matrix electrode array. Block1230 then produces the resulting position data 1232.

In one or more embodiments, the NoiseTouch system does not includefacilities for transmission of signals. Signal transmission facilitiescan be omitted because NoiseTouch senses environmental signals, and doesnot need to transmit signals in order to sense environmental signals.Since the receiving hardware is designed to accept EMI, it is resistantto interference from EMI sources. In addition, the system does not emitspurious signals as a result of its activities outside of what may benormally expected from any electronic circuit, simplifying compliancewith EMI regulations and design of noise-sensitive electronicspositioned nearby. An additional benefit is the power savings from thisdesign. On one hand, there is direct savings from not having to transmita field. Additionally, the system benefits from a simplifiedarchitecture, which means there is simply less electronics to power tobegin with. Additionally, since there is no need to carry out extensivenoise rejection in hardware, there is additional savings from thereduction of complexity on that front as well.

FIG. 13 illustrates an example process that determines hand poses orpositions. The EMI conducted by the body is coupled capacitively to theelectrode via the electric field surrounding the body. As an example,the process can determine when the user is holding or touching thescreen from the left or right side, which is pose information. An ADC1306 converts the analog input signal from the amplifier 1302 to adigital signal. By appropriately adjusting the gain of the system atblocks 1308 ad 1304, NoiseTouch can detect approach of a body part at adistance. As such, it is possible to distinguish when the user ishovering over the touch surface without making physical contact.Additionally, because of the speed afforded by the NoiseTouch system,the electrodes can be continuously scanned at multiple gain settings atblock 1310, allowing for simultaneous detection of hover and touch.Multiple gain setting scanning may be used, for example, to allow forpalm or inadvertent contact rejection, detection of holding pose(one-handed vs. two-handed, left vs. right handed, etc.), and the like.Block 1312 compares the signals read at different gains. Bock 1314 usespose heuristics to determine pose data, such as the pose of a hand.Block 1318 uses the results of the signal comparison 1312 to determinehover position.

The multi-gain surface scanning can detect the pose of the hand as theuser holds a device that contains a NoiseTouch sensor. Multi-gainscanning provides different sensing depths, with resolution decreasingwith the increase of the gain. At high gain, it can sense more distanceobjects, but does not determine position as exactly as when low gain isused. For example, multi-gain scanning can enable the system todistinguish right-handed pen input from left-handed pen input bylocating the position of the hovering hand relative to the contactposition. The location can be determined using a higher gain surfacescanning setting to sense the approximate position of the hand that ismaking contact. Multi-gain scanning can also help to sense whether oneor two hands are hovering, which from the sensing perspective willproduce, respectively, one or two sensed “blobs” at medium gain, or asmall or large “blob” at high gain. Since the sensing field at highgains extends out from the device to some distance, it is also possibleto detect how a device with a NoiseTouch screen is being held relativeto the location of the screen.

In one or more embodiments, gestures that are part of “touch” (e.g.,multi-gain hover and the like) can be separated from how the machine canreact to the presence of hover. For example, if a user is holding thephone in their right hand, the keyboard can automatically shift itstouch points to the left so that the user can type more easily. Also,controls can appear on a tablet closer to the hand holding the tablet(or, alternatively, on the other side of the table, so that touching thetablet with the free hand is easier). In one aspect, hover can be acontextual cue for the software.

FIG. 14 illustrates an example method of separating touch and stylusdata. Similar to the NoiseTouch techniques described above in FIG. 12,an input signal can received by an ADC 1420 and characterized by a noisecharacterization block 1422. Noise separation is performed by a modifiednoise separation block 1424, and position data 1428 is determined by aposition estimation and linearization block 1426. While the descriptionabove focuses on the appendages of a user, it should be noted thatNoiseTouch can function equally well with conductive orpartially-conductive objects. As such, it is possible to fashion styli,pens, or other devices 1410 that can be detected by NoiseTouch. In suchcases, the design of the stylus 1410 can include passive reactiveelements, such as inductors or capacitors (or a combination of passiveelements), which imprint the noise signal with a specific signature thatis detectable by the position estimation 1426 and noise characterization1422 blocks. Detecting the specific signature enables NoiseTouch todiscern between the presence of a stylus 1410 and a finger. Therefore,separate finger position data 1428 and stylus position data 1430 can begenerated by the position estimation and linearization block 1426.

FIG. 15 illustrates detection of the signal modification characterizingthe modification of ambient noise by the contact by a stylus or pen1510. FIG. 15 illustrates example signals at different points of theanalog front-end portion of the system shown in FIG. 14. The EMI sourcesemit an EMI signal. The stylus 1510 emits a signal, and another signalis received by the electrode 1418. The signal received by the electrode1418 is different from the EMI received by the electrode at Z_(air) whenthe stylus is not in contact with the insulator 1416.

An alternative implementation of the device may produce a certain amountof controlled generalized EMI from the device which is then used todetect position in areas where sufficient environmental EMI may not beavailable. This capability may be automatically switched on by theautomated gain control systems once the levels of environmental EMIdrops below a pre-programmed or dynamically selected threshold. TheNoiseTouch system may be tuned to specifically use theregulatorily-allowed EMI emissions of the device exclusively, thusrejecting other sources of noise. This increases the robustness of thedevice since the EMI profile need not be dynamically characterized.

The NoiseTouch system may use one instance of the above-mentionedhardware for each touch position, or it may use a continuous largerelectrode and estimate the position based on the distance-dependentchange in signal through the electrode. The change may be caused bymaterial properties of the covering material, resistance of theelectrode body, reactive impedance of the electrode, or any othermethod. In this way, NoiseTouch can distinguish position at a higherresolution than the resolution of its electrode structure.

NoiseTouch can be configured for single or multiple touch points, andadditionally can be configured for either continuous position sensing(such as a phone or tablet), or discrete position sensing (such as abutton). In the latter application, the strengths of the system remainin force, and make the system practical to use in many scenarios whereenvironmental noise or contamination may be an issue, such as inautomotive or marine uses, in factory floors, etc. In such cases,NoiseTouch can provide the benefit of a robust input solution withoutthe need for additional precautions necessary for traditional capacitivesensing.

FIG. 16 illustrates an example process of passively sensing theenvironment and context of a user. NoiseTouch can continuously sense andcharacterize the environmental EMI, and this capability can be used topassively sense the environment and context of the user. For example, athome the user may surrounded by EMI from the TV, mobile phone, andrefrigerator, while at office the user may be surrounded by the EMI fromthe desktop computer, office lighting, and office phone system. When theuser makes contact with the NoiseTouch system, e.g., to awaken or unlocktheir device, the NoiseTouch system can capture this characteristic dataand compare it to an internal database of noise and environments, usingrelevant similarities to deduce the user's location.

In the process of FIG. 16, an input signal is fed from a signalacquisition system 1602 to a Noise Characterization module 1604. Block1604 performs noise characterization to determine a current noiseprofile 1610. Block 1604 breaks the signal down into bands (e.g., usingFFT or the like), and analyzes both the magnitudes of signals indifferent signal bands, as well as the time-domain change in thosemagnitudes. The signals to be used for positioning are fed to block1606. Block 1616 performs estimation and linearization to generateposition data 1608, as described elsewhere herein. From user input 1606,as well as automatic sensing 1618, such as GPS, WiFi positioning, etc.,block 1620 determines whether the device is in an environment ofinterest. If so, the current noise profile is stored in the Environmentand Context Database 1622. The current profile and the entries in theDatabase are used by the Environment and Context recognizer 1612 todetect when the environment or context is encountered again, and whenrecognized again, events are generated accordingly.

FIG. 17 illustrates examples of noise contexts that can be passivelysensed. Different rooms in a house or office can have different noisecontexts. For example, a break room may include EMI from the coffeemachine, while a meeting room may include EMI from a large TV orprojector, as shown in FIG. 17. The device can then use the contextestimates to make certain functionality easily accessible. For example,the device can automatically print queued documents from the user whenthe user approaches the printer, or allow control of the projector whenthe user is in the same room. The user may additionally configurecapabilities on a per-area or per-context basis to help streamlinetasks. The noise characterization can be performed based on externaldevice activity 1702, such as whether a TV or lights are on or off. Thecontext of interest can be based on automated context recognition 1704or user input 1706. The automatic context recognition 1704 can determinethat the context is “leaving the kitchen”, “in the bedroom”, or“driving”, for example. The user input can be “watching TV”, “reading inbed”, or “washing clothes”, for example. Based on these factors, theenvironment and context data 1708 is generated. and used as input forcontext-relevant or context-dependent automation services 1710.

FIG. 18 illustrates an example process of using the context sensingsystem to communicate with a device having a NoiseTouch sensor. A devicewishing to communicate may emit a capacitive signal 1804 by applying avoltage to a conductive surface, including the metal frame or shieldingof the device itself. This signal 1804 is combined into theenvironmental EMI field 1802 and received by NoiseTouch. The signal canbe encoded by a user or stylus at block 1808, and received by anelectrode and ADC at block 1808. Noise characterization can be performedat block 1810, and position estimation and linearization can beperformed at block 1812 to generate position data 1814. A signaldetection system 1818 can be used to search for such signals, possiblywith additional information 1816 about the context which allows anarrowing of search parameters. Thereafter, the noise signal is filteredat block 1820 to only include the bands where transmission is takingplace, and the signal is demodulated at block 1822 to receive 1824 data.Such data may be used to uniquely identify a device (for example, toallow immediate and direct control of a nearby appliance), or to sendcontextual data (such as the time remaining for an oven to heat up, orthat a refrigerator door has been left open). Such communication may bemade bidirectional, such that a device which does not include theposition-sending capability may nonetheless include a NoiseTouchelectrode for the purposes of receiving context and incoming data.Therefore a non-touch-enabled device (such as a microwave) may receiveNoiseTouch-based communication from a nearby device for the purposes ofcontrol or query of functionality.

Example scenarios in which environmental sensing can be used includechanging a phone's home screen depending on sensed context, changing aphone's home screen depending on sensed context, sending user locationto external devices using context sensed by the phone, targeted sensingof activity of external devices, and monitoring energy consumption. Thesensor system may be located on a device such as a watch or Fitbit-typedevice that is word by the user. The sensor system can also be on alaptop or a TV. For example, when a user enters a house, the phonedetects the noise signature of the house and provide a set ofapplications on the home screen that are dedicated to home control, e.g.Alarm control, TV, Audio system, etc. A phone's home screen can bechanged depending on sensed context. Upon the user entering the house,the phone detects the noise signature of home and provides a set ofapplications on the home screen that are dedicated to home control, e.g.Alarm control, TV, Audio system, etc. For example, a tablet orsmartphone can display up a home screen page that contains musicapplications when a headphone is plugged in. Likewise, when the user isat home, the controls for various appliances, lighting systems, TV andother electronics, home HVAC controls, etc., can be brought up on aspecial page of the interface that makes access much more convenient. Inanother example, the home can be enabled to provide applicationdedicated to the control of devices in each room, privileging TVcontrols when in the living room, and timer when in the kitchen forexample. When a user moves from room to room in the house, the homescreen can be changed depending on the sensed environmental context.This technique can be applied on a per-room basis. For example, the usermay customize a page that displays business-related applications such asemail and business document management software when the user is in thestudy, the TV remote and current TV schedule in the living room, and thebaby monitor, security system, and AC controls in the bedroom. These mayassociations can be designed to be customized and managed by the user.

A user's location can be sent to external devices using context sensedby the phone. For example, the phone detects the current room the useris in, and sends the information to the devices in the current room.Lights can be turned on when the user carrying his phone enters a givenroom, and turns off when leaving it; A preset profile, e.g. certainmusic and lighting conditions can be started automatically when the userenters the living room; Alarm could be de-activated when entering thehouse, and so on. For example, the system may notify the TV when itdetects the user has moved away. At that point, the TV may turn off apower-consuming display panel, but leave the sound on, saving energy.The air conditioning may go into a power-saving mode likewise when theuser is away, and quickly cool a room when the user enters. The user mayconfigure the devices to act in a particular way based on his or herpresence or absence from the vicinity. In one or more embodiments, Ifthe TV is on, the phone may look up favorite programs the user selectedpreviously, and tell the user that a particular channel is showing hisfavorite show.

Noise detection can also be used to target activity sensing of specificexternal devices, such as TV, lights, audio system etc. For example, aphone can detect that lights are left on when in the hallway beforeleaving a place, and notify the user. As another example, a phone candetect that a television s switched on and can provide recommendations,and the like. To perform energy consumption monitoring, noise detectioncan sense the overall noise level of a home to monitor the activity ofelectronic devices and give a sense of global energy consumption. Usingsignal processing on the global noise level, energy monitoring can alsobe targeted and device-specific. All electronics, when active, canproduce more EMI than when off. By sensing the overall changes in bulkEMI, the system may determine when the user is generally using more orless energy, and provide overall feedback without necessarily detectingparticular devices or knowing anything particular about those devices.Therefore, when the user is in a room, the sensing system can detect ifthe lights are or not. When the user moves to a different area as notedby the system based on a change in the EMI environment, the system cannotify the user that they left the lights on. This may be additionallygated by particular locations, such that it only applies to home,office, or otherwise. Note that on one or more embodiments thistechnique requires no special instrumentation of the lights or otherinfrastructure, and thus can be easily used with legacy unaugmentedlocations.

In addition, NoiseTouch and hover can be used to detect a single airtouch/tap, dual air touch/tap, multi-finger air touch/tap, adjacentfingers hovering, or hovering thumb versus fingers. Furthermore, motionusing hover can be detected such as, for example, zoom in, zoom out,panning, dragging, scrolling, swipe, flick, slide, rotation clockwise,or rotation counterclockwise. In addition, portions of content under thehovering object can be magnified or previewed. Also, objects can berecognized by detecting the conductive parts of the object. Furthermore,when holding insulating objects, NoiseTouch allows for detecting thetool angle, and the position of the hand relative to the object.

FIG. 19 illustrates an example architecture of a TriboNoiseTouch system.In one or more embodiments, the TriboNoiseTouch techniques disclosedherein are based on a combination of the TriboTouch and NoiseTouchtechniques. In one or more embodiments, NoiseTouch uses the noise thatis conducted by the human body and picked up by the electrodes of thetouch sensor to detect the position of the user's touch. In one or moreembodiments, TriboTouch uses the charge displacement that occurs whentwo objects come in contact with each other. By measuring thisdisplacement, TriboTouch can detect contact of the sensitive surfacewith any material. This is done using a sensing surface similar tocapacitive sensors in use today and requires no physical displacement(as resistive screens do).

In one or more embodiments, TriboNoiseTouch combines the capabilities ofTriboTouch and NoiseTouch using the same hardware, electrode geometry,and processing architecture. Therefore, the TriboNoiseTouch system hasthe capacitive touch features of NoiseTouch, and is also capable ofsensing contact with a wide variety of materials using TriboTouch.TriboNoiseTouch opportunistically uses each methodology to offerimproved capabilities, further improving the speed of contact detectionover NoiseTouch, while providing non-contact and bulk contact (e.g.,palm contact) sensing. TriboNoiseTouch uses environmental noise andsurface interaction. TriboNoiseTouch can thus be immune to EMI, and neednot emit an electric field. TriboNoiseTouch can sense the contact ofnon-conductive materials. Additionally, TriboNoiseTouch uses acombination of two physical phenomena to detect touch and providerobustness, speed, and differentiation of contacts by differentmaterials (e.g., finger vs. stylus). The combination of NoiseTouch andTriboTouch technologies into a single panel can reduce complexity andprovide savings in energy, and reduce hardware resource usage.

While the sources of signals for noise and triboactive measurement aredifferent, the characteristics of the signals have similarities. Bothsignals are ordinarily coupled to the electrode capacitively via anelectric field, and are therefore ordinarily amplified by ahigh-impedance amplifier. This allows the hardware for triboactive andnoise-based position sensing to be economically combined into a singleTriboNoiseTouch system. The TriboTouch and NoiseTouch techniques can becombined using time multiplexing or space multiplexing. For example, afull panel reading can be performed with TriboTouch, and then withNoiseTouch, or we some of the electrodes on a panel can be used forTriboTouch, and others for NoiseTouch, with optional switching ofelectrodes between TriboTouch and NoiseTouch for more continuouscoverage.

Referring to the example TriboNoiseTouch system shown in FIG. 19,environmental noise sources 1902, such as power lines, appliances,mobile and computing devices, and the like, emit electric fields thatcontribute to the environmental electromagnetic interference (EMI, orcolloquially, electronic noise). The human body 1904 is a slightconductor, and thus acts as an antenna for these signals. When the body1904 closely approaches the electrode 1906, e.g., when the body 1904 ishovering over or touching a touch panel, this signal is capacitivelycoupled to the input electrode 1906. At the same time, the contact ofthe body or another object with the touch surface causes a triboelectricsignal 1908 to be produced. Both signals are capacitively coupled to theelectrode. A high-impedance amplifier or electrometer 1910 detects theincoming signals, and an analog to digital converter (ADC) 1912subsequently converts this signal to digital form. These components mayhave additional switchable characteristics that aid in the separation ofthe two signals.

The signal is processed by a processing system 1916, which can beimplemented as hardware, software, or a combination thereof. Theprocessing system 1916 can include a calibration, which can be done atstartup, and whenever internal heuristics determine that the signal isbecoming intermittent or noisy. This is done, for example, bycalculating mean and variance, and ensuring these values remain within arange. Deviations of the mean value may lead to gain adaptation, whileexcessive variance may cause the selection of a different noise band.

The processing system 1916 has two stages of execution. For thetriboactive signal, the processing system 1916 characterizes the noiseat block 1920 and adapts the gain at block 1918 so that the signal doesnot overwhelm the amplifier. This stage can be done separately fortriboactive and noise signals, in which case the processing system 1916characterizes the noise at block 1926 and adapts the gain at block 1924for the noise signals. Additionally, offsets in the readings caused bycharges adhered to the insulators or nearby objects can be offset fortriboactive signals at block 1922. The initial conditions are calculatedduring the initialization phase. Noise source selection is performed atblock 1928.

After initialization is complete, the data processing portion of thesystem begins at block 1930. Block 1932 selects the measurement to make,and block 1934 separates the signals by applying initial filtersspecific to the signals required. The characteristics of the filters aresuited to the selection of noise signals, as well the means ofinterleaving the two types of measurements. For noise signals, theprocess continues gain adaptation at block 1936 and rejects unwantedsignals at block 1938. For triboactive signals, the gain and offset areadapted to compensate for environmental drift at blocks 1940 and 1942,respectively. The gain adaptation information is fed back to gaincontrol block 1914 to control the high-impedance amplifier 1910, so thatthe signal from the amplifier 1910 remains within the range of the ADCblock 1912. The outputs of both signal paths feed into the opportunisticposition estimation and linearization block 1944, which uses the mostreliable and time-relevant features of both measures to calculateposition estimates 1946.

FIG. 20 illustrates an example method of separating triboactive datafrom noise data. As shown, during initialization a characteristicprofile of noise and triboactive signals is created at blocks 2002 and2008, respectively. At runtime, signal separation block 2014characterizes the triboactive signal in time and frequency domains,indicating what signals come from triboactivity. The remaining signal isthan analyzed by band and appropriate bands are selected for the noiseanalysis at block 2016.

The system starts with an initialization of the system where wedetermine (possibly offline) specific initial signal bands. Signalseparation may operate in the time or frequency domain, and may be doneby filtering specific frequency bands from the combined signal. Atruntime, the signals are separated according to the initializationcharacteristics determined, and the data is separated into independentstreams for processing. The band selection may be dynamically changedbased on location, signal strengths, etc.

In one or more embodiments, the TriboNoiseTouch system does not includefacilities for transmission of signals. Signal transmission facilitiescan be omitted because TriboNoiseTouch senses signals in the environmentas well as to the contact itself, and does not need to transmit signalsto sense environmental signals. Since the receiving hardware is designedto accept EMI, it is resistant to interference from EMI sources. Inaddition, the system does not emit spurious signals as a result of itsactivities outside of what may be normally expected from any electroniccircuit, simplifying compliance with EMI regulations and design ofnoise-sensitive electronics positioned nearby. An additional benefit isthe power savings from this design. For example, there can be directsavings from not having to transmit a field. The system benefits from asimplified architecture, which means there is simply less electronics topower to begin with. Additionally, since there is no need to carry outextensive noise rejection in hardware, there is additional savings fromthe reduction of hardware complexity.

FIGS. 21-23 illustrate example TriboNoiseTouch processes for identifyingtriboelectricity-related events and noise-related events. Three exampleprocesses for sequencing TriboNoise event-sensing are described herein.The process of FIG. 21 identifies triboelectricity-related events, thenidentifies noise-related events (i.e., TriboTouch first). In one or moreembodiments, the system can trigger the NoiseTouch subsystem when theTriboTouch portion of the system has received no signals after a periodof time has elapsed. Each TriboTouch event transmits a touch-event ormaterial classification event when detected.

The process of FIG. 22 identifies noise-events, then identifiestriboelectricity events (i.e., NoiseTouch first). In one or moreembodiments, in the NoiseTouch-first setup, a timer can be used to resetthe noise gain settings after no interruption has been sent by aTriboTouch-recognition pipeline after a given amount of time has passed

The process of FIG. 23 is an example sweep process that acquires a wideband signal and parallelizes triboelectricity sensing and noise sensing.The sweep process of FIG. 23 can be used, for example, whenprioritization is to be set at a higher level, e.g., at applicationlevel. For example, a painting application may be more closely relatedto triboelectricity-based sensing, while location/context dependentapplications may be more closely related to noise-based sensing.

The choice regarding the relative prioritizations of TriboTouch andTriboNoise can be device- and application-dependent. Thetriboelectricity-first approach is well-suited for applications wheretouch surfaces are used heavily by the user, while the “noise-first”approach is well-suited for more general application devices, such asmobile devices, where context sensing on and above the surfaceinteraction can be used simultaneously. Similarly, contextdependent-applications are likely to privilege noise-sensing, whiledrawing, painting, and other direct manipulation applications are likelyto privilege triboelectricity-sensing.

By combining noise and triboactive measurements, it is possible todetect materials that are not sufficiently conductive to be visible tonoise-based or capacitive measurements. In addition, the characteristiccontact reading involved in triboactive measurement obviates the needfor extensive threshold estimations for detecting touch. This means thatthe system is able to react to short contact events such as the userusing a stylus to dot the lowercase letter “i”. The combination of thesystems also allows for the detection of body parts and hand-heldinstruments such as styli. In such cases, the stylus can simply be madeof an insulator that is “invisible” to noise-based measurements, whichallows the system to detect whether a contact is made by, for example,resting the wrist on the touch surface, or by the stylus held in thesame hand.

FIG. 13, described in part above, illustrates a process ofsimultaneously detecting hand pose information and hover position.TriboNoiseTouch systems can determine when true contact has occurred,thus preventing phantom readings from fingers held close to the touchsurface from accidentally invoking commands. This is a side effect ofthe fact that triboactive signals are only generated by direct contact.However, it is also possible to simultaneously detect hovering as well,thus presenting additional means of interaction. Since the EMI beingconducted by the body is coupled capacitively to the electrode via theelectric field surrounding the body, by appropriately adjusting the gainof the system, NoiseTouch is capable of detecting approach of a bodypart at a distance. Because of the TriboNoiseTouch system's speed, itcan continuously scan the electrodes at several gain settings, allowingfor simultaneous detection of hover and touch. This may be used, forexample, to allow for palm or inadvertent contact rejection, detectionof holding pose (one-handed vs. two-handed, left vs. right handed, andso on).

The process shown in FIG. 13 can take readings from the electrodes witha variety of gain settings, usually above the nominal setting used todetect contact. At higher gain, weaker and more distant electric fieldsare detected. By stacking up these weaker images at different gains, thesystem can detect what is near the sensing surface. For example, given atouch gain setting G, a finger hovering above would be detected atsetting G+1, some of the knuckles at setting G+2, some of the hand andpalm at gain setting G+3, and so on. Of course, further away objectscannot be “seen” by the sensor as well, but we can gather someinformation that then tells us if a user is hovering, which hand isholding the device, etc.

In one or more embodiments, TriboNoiseTouch hardware enables thedetection of context, hover, contact, and material identification.Context dependent touch applications can then be provided. After contextis sensed, specific touch applications and multi-material applicationscan be triggered, e.g. a remote control application when entering livingroom, or drawing application when entering the office. In addition,context can be used while the device is in standby to detect whatapplications and controls should be available to the user. Moreover,when TriboTouch is used to detect contact, the NoiseTouch can be used asbackup or shut down completely to save power. TriboNoiseTouch can alsoprovide high precision input. Using the integration of both TriboTouchand NoiseTouch, contact sensing coordinates can be used for highprecision input in, e.g. technical drawing applications, or ininteraction on very high definition displays.

An alternative implementation of the device may produce a certain amountof controlled generalized EMI from the device which is then used todetect position in areas where sufficient environmental EMI may not beavailable. This capability may be automatically switched on by theautomated gain control systems once the levels of environmental EMIdrops below a pre-programmed or dynamically selected threshold. Thislogic may take into account the demands placed on the system, such thatwhen hovering functionality is not necessary, the system can switch tousing triboactive mode exclusively, maintaining sensitivity whileexcluding detection of contact type. The noise-sensitive component ofthe system may be tuned to specifically use the regulatorily-allowed EMIemissions of the device exclusively, thus rejecting other sources ofnoise. This increases the robustness of the device since the EMI profileneed not be dynamically characterized.

The TriboNoiseTouch system may use one instance of the above-mentionedhardware for each touch position, or it may use a continuous largerelectrode and estimate the position based on the distance-dependentchange in signal through the electrode. The change may be caused bymaterial properties of the covering material, resistance of theelectrode body, reactive impedance of the electrode, or any othermethod. By this means, TriboNoiseTouch may be able to distinguishposition at a higher resolution than the resolution of its electrodestructure. TriboNoiseTouch may be configured for single or multipletouch points, and additionally may be configured for either continuousposition sensing (such as a phone or tablet), or discrete sensing (suchas a button or slider). In the latter application, the strengths of thesystem remain in force, and make the system practical to use in manyscenarios where environmental noise or contamination may be an issue,such as in automotive or marine uses, in factory floors, etc. In suchcases, TriboNoiseTouch can provide the benefit of a robust inputsolution without the need for additional precautions necessary fortraditional capacitive sensing. Additionally, the system remainssensitive even when the user is wearing a bulky glove or using anon-conductive tool to trigger the control, allowing greater flexibilityin terms of method of use and environmental contamination orinterference.

TriboNoiseTouch's features that continuously sense and characterize theenvironmental EMI can be used to passively sense the environment andcontext of the user. For example, at home the user may be surrounded byEMI from the TV, mobile phone, and refrigerator, while at office theuser may be surrounded by the EMI from the desktop computer, officelighting, and office phone system. When the user makes contact with theTriboNoiseTouch system, perhaps to awaken or unlock their device, theTriboNoiseTouch system can capture this characteristic data and compareit to an internal database of noise and environments, using relevantsimilarities to deduce the user's location. This process is illustratedin FIG. 16. Note that different rooms in a house or office may have verydifferent noise contexts. For example, the break room may include EMIfrom the coffee machine, while the meeting room may include EMI from alarge TV or projector. The device can then use the context estimates tomake certain functionality easily accessible. For example, it mayautomatically print queued documents from the user when the userapproaches the printer, or allow control of the projector when the useris in the same room. The user may additionally configure capabilities ona per-area or per-context basis to help streamline tasks.

The triboactive portion of the system produces high-resolution databased on individual micro-contacts with the surface of the touch sensor,while the noise-based sensing subsystem produces a blob around the areaof contact or hover as well as a “shadow” of the hand hovering over thesurface (see FIG. 24). These three types of data can be combined tocreate additional capabilities that are not available to either sensingmodalities in isolation.

The accuracy of finger contact can be enhanced by using a combination ofTriboTouch and NoiseTouch type sensing. TriboTouch-type normally willproduce a cloud of contacts around a finger contact due to themicro-texture of the finger interacting with the sensing electrodes. Thenoise data can be used at the same time to give an accurate position forthe centroid of the contact, thus allowing the tribo data to be cleanlysegmented to be inside the noise blob. The exact tribo contact positionscan them be used to estimate the shape, size, and intended exact contactposition. FIG. 25 shows the method for doing this refinement.

Even if the touch sensing surface has not been treated to sensematerials, or such algorithms are not active, a finger contact can bedetected and isolated from a non-conductive pen contact. Since the penis not conductive, it will not register in the noise-based sensing,while finger contact will produce both types of contact data. This canbe used to control different refinement algorithms based on pen orfinger contact, and to allow the simultaneous use of fingers and pens.The algorithm is shown in FIG. 26. The system provides both enhancedposition based on type of contact, as well as notification of the typeof contact event.

The pen or hand pose can be estimated by detecting the hover shadow ofthe hand making contact or holding the pen. The overall shape of thehand, as well as the shape of the hand while holding a pen can bedetected by using a pattern matching algorithm or heuristic, and thiscan be used to detect whether a contact is made by the left or righthand, as well as estimate of pen or finger tilt. Tilt is calculated byestimating the point where the stylus or pen is held, and the actualpoint of contact. The same approximate measurement can be made aboutfinger contact and finger angle. The algorithm is shown in FIG. 27.

Additional data can be made available to client programs to detectover-screen gestures, as well as disambiguation of left and right-handedcontact. This can allow for example control of tool type with one handwhile the other is used for manipulation, without two contactsaccidentally triggering pinching gesture heuristics.

As noted previously, the TriboTouch system can be used to detect thematerial making contact by examining the differences in chargedisplacement caused by various materials. Noise signals are transmittedthrough conductive and resistive object. As a result, it can helpclassification of materials done by TriboNoiseTouch hardware by quicklydiscriminating materials depending on their conductivity. For example,when interacting with the TriboNoiseTouch enabled display, the tip ofthe pencil could be detected to automatically trigger the drawing tool,while using the eraser of the pencil will trigger the erasing function.In this scenario, the NoiseTouch hardware will be able to detect the useof the tip of the pencil because it is conductive and will trigger bothnoise and tribo signals. On the other hand, the eraser will onlygenerate tribo-electric signals.

TriboNoiseTouch can be configured such that NoiseTouch is triggered onlyafter contact has been sensed by the TriboTouch hardware. This systemwill only focus on contact-based interaction, such as touch and peninteraction, and will not be able to sense interaction above the surfacesuch as hover. However, this will enable power savings and prevent bothTribo and Noise hardware (and their respective signal processingpipelines) to actively wait for interaction events. While the same frontend is used for both, the reduction in calculations reduces the dynamicpower usage of the digital logic used to run the triboactive andnoise-based position calculations.

While TriboTouch sensing can provide high resolution stylus sensing,TriboNoise can be used to detect a specifically designed stylus thatfeatures buttons to trigger menus and functions. The stylus will usetribo and noise signals together to detect position, where for exampletriboelectric signals will enable sensing contact, release and draggingstates, while sensing noise will help to recover position duringdragging states, hold, as well as get information from button presses(see FIG. 28). The core of the stylus consists of an antenna thattransmits noise signal to the panel when the pen is in contact with thesurface. The button enables adding to the antenna path a filteringcircuit that will affect the noise signal in a predictable way, byadding a complex impedance or nonlinear behavior (like a diode) to thesignal path. By analyzing the signal injected into the panel by the pen,the system can detect if the button has been pressed or not. In the caseof a change to impedance caused by a button, a change in phase oramplitude at certain frequency will be the indicator of a button press.In case of a diode or other non-linear element, harmonics of a certainfrequency will be sensed when the button is pressed due to clipping orshaping of the incoming noise signal.

Because triboelectric charging occurs when objects make or breakcontact, it is possible to detect these events more precisely usingTriboTouch alone or in combination with NoiseTouch or other sensingmethods. By contrast, NoiseTouch alone uses a threshold value (that maybe adaptive) to determine when contact occurs. Because the tribochargedistribution and polarity depend on the direction of motion (toward,away from, and along the surface), these events can be distinguishedfrom hovering or near-contact events. This allows a finer control overthe range of values considered for hovering, and thus improves thedynamic range for hover sensing (see FIG. 29).

While TriboTouch is good at detecting contact, separation, and motion,it cannot detect static objects. Therefore it is complemented by the useof NoiseTouch to detect position and shape of conductive objects duringlong static contacts.

Another scenario is the simultaneous use of a nonconductive stylus,brush, or other object detected solely by TriboTouch in combination withfinger gestures detected by both TriboTouch and NoiseTouch. Anapplication can distinguish between the fingers and the stylus becauseof the differences in their TriboTouch and NoiseTouch characteristics,and therefore process their corresponding events differently. Forexample, stylus input can be used to draw and brush input to paint,while finger input can be used to manipulate the image. For example,this allows the user to zoom using hover and simultaneously use plasticstylus to draw; to adjust the drawing space as the user is drawing; toscale with fingers while drawing with stylus; or to control a drawingparameter such as brush color intensity with hover while simultaneouslydrawing with a stylus.

By patterning conductive and non-conductive materials onto an object,information may be encoded to allow recognition of the object. Forexample, the bottom of a game piece may be encoded with a pattern ofmaterials that allow its identity and orientation to be detected.

FIG. 30 illustrates example single-touch electrode components, which areone type of electrode configuration that can be used with theTriboTouch, NoiseTouch, and TriboNoiseTouch techniques disclosed herein.Other electrode configurations can also be used. In particular, theelectrode types disclosed herein include (1) single-touch electrodes,(2) dual-touch electrodes, (3) array multi-touch electrodes, includingthe multiple-electrode configuration shown in FIG. 34, (4) continuouspassive position sensing, (5) continuous two-dimensional passiveposition sensing, (6) dielectric-encoded passive position sensing, (7)continuous passive position sensing using an array of non-linearelements, and (8) spatially-distributed coordinate encoding. Types(1)-(7) can be used with any of TriboTouch, NoiseTouch, andTriboNoiseTouch. Type (8) can be used with TriboTouch orTriboNoiseTouch. Any of these valid electrode-detection combinations(e.g., a combination of one or more of the electrodes (1)-(8) and one ofthe TriboTouch, TriboNoise, and TriboNoiseTouch detection techniques)can be used with the same analog front-end, such as the analog front-enddescribed above with reference to FIG. 3.

Referring again to FIG. 30, a single-touch electrode can be designed toact as a switch, or can be arranged in an array as an element of alarger surface. A single-touch electrode with these components is shownin FIG. 30. The components include an insulator layer and senseelectrodes. The shield electrode and ground shield electrodes may beomitted at the cost of degraded performance, though performance mayremain sufficient for touch detection. The shield electrode may beinter-digitated with the sense electrode such that the distance betweenthe lines of the two electrodes is minimized. This may be done withsimple inter-digitation, or via the use of a space-filling curve. Aspecific instantiation is the use of an inter-digitated Hilbert curve.The use of the inter-digitated electrodes is used to reduce theparasitic capacitance of the electrode relative to the environment byactively driving the electrode using the output of the high-impedanceamplifier of the sensing system. An additional shield electrode may beused to reject input to the system from the direction opposed to thefront of the surface. This prevents spurious detection of contact due toEMI produced by nearby electronics, such as the display in the case of atransparent touch surface application such as a tablet.

FIG. 31 illustrates two electrodes (2602 and 2604) in an exampleinterleaved pattern. In the interleaved electrode, only the shield andpickup electrodes are shown. Electrodes may be used interchangeably forpickup or shield. This is a simple example of interleaved patterns, andthe conductive portions of the electrodes may be more complexlyintertwined.

FIG. 32 illustrates a row-column electrode grid that can be used todetect position of two touch points. Note that unlike capacitive touchsensors, row-column configurations do not directly offer the ability tosense multiple touch positions, since the electrodes are used as senseelectrodes, and in the triboactive and noise-based sensors, transmitelectrodes may not be present. In this configuration, two touch pointscan be distinguished, though their exact positions can be lost. However,this is sufficient for common gestures such as two-finger tap orpinch/expansion gestures. Other example gestures can be a wave or sweepmotion made over the screen without contact, or a hovering motion over acontrol (which can elicit a highlighting feedback).

FIGS. 33 and 34 illustrate array multitouch configurations usingsingle-touch electrodes in a grid. Each electrode individually picks upcontact near it. However, since electric fields as well as the chargecloud produced by triboactivity expand outward from the source chargedeflection, the field can be detected by nearby electrodes as well, asshown in FIG. 34. As a result, the position of the contact can beinterpolated between the electrodes that receive the signal. Likewise,because capacitive coupling takes place at some distance, thenoise-based sensor can detect the presence of a hovering conductive bodysuch as the user's finger, allowing for hover sensing.

FIG. 35 illustrates an example of continuous passive position sensingusing a resistive sheet electrode. For continuous passive positionsensing, a sheet electrode with some known uniform resistance per unitof area can be used alongside pickup electrodes that are placed on thisresistive sheet 3002. The configuration shown in FIG. 35 involves alinear sensor with two pickup electrodes. Continuous passive positionsensing is performed by detecting the apportionment of chargedisplacement from a contact. When the impedance of the sheet matches(approximately) the impedance of the system, the value sensed at eachpickup is some function of distance to the contact charge cloud. Bycharacterizing and linearizing the readings from the pickups, it ispossible to detect the position of the contact continuously at anyposition up to the accuracy and precision of the digitizationelectronics and the noise characteristics of the system itself. Thisapproach leads to simpler electronics and a simpler patterning of theprimary touch resistive sheet, which in turn leads to lower cost andcomplexity. The position of contact can be calculated based on theproportion of output each pickup relative to the total signal captured.Conversely, a global noise pickup layer may be laid under the resistivelayer to sense to total amount of charge injected into the surface, thusallowing a direct comparison.

FIGS. 36 and 37 illustrate an example of continuous two-dimensionalpassive position sensing. The passive position sensing technique shownin FIG. 35 can be extended to two dimensions, as shown in FIG. 36. Thetwo-dimensional technique can sense n points of touch 3104 from thesignals induced in a resistive sheet 3102 with a known distribution of mpickup points 3106. The inputs to the touch surface at time t are nindependent voltages Vi(t) at coordinates (xi, yi) 3212 for each pointof touch, as shown in FIG. 37 Voltages are measured at m known pickuppoints 3204, 3206, 3208, 3210, 3212 on the edges of the resistive sheet3102. By approximating the resistive sheet as M×N network of resistorsand the use of already-known methods, the resistance between a pickuppoint and a touch point may be found. The relationship between theresistance between a given pickup point and a touch point is used todetermine the voltage at a given pickup point. The resulting equationrepresents the dependence of the voltage level at a pickup location onthe coordinates and input voltages at the touch points. From this systemof equations for voltage levels at pickup points, the touch pointcoordinates (xi, yi) and input voltages Vi(t) are found. The number ofrequired pickup point locations m is at least 3 n; a larger number ofpickups may be used to reduce errors due to numerical approximations andmeasurement error. The known distribution of pickup points and thenon-linearity of the resistive sheet allow separation of the touchpoints and their distribution. This method can be further generalizedfrom points of contact (x_(i), y_(i)) to points of hover (x_(i), y_(i),z_(i)) by solving for a third unknown coordinate. This generalization topoints of hover increases the minimum number of pickups m from 3 n to 4n.

FIGS. 38-40 illustrate example electrode-sheet configurations. Theelectrodes can be designed with pickups and resistive sheet on differentlayers, or on the same layer, as shown in FIG. 38 and FIG. 39respectively. FIG. 38 shows the pickups 3306 and resistive sheet 3302 asdifferent layers, separated by pickup contacts 3304. Additionally, toincrease the resolution of contact readouts, several of these patchesmay be arrayed next to each other with minimal gaps between them forpickup electrodes to create a single layer high-resolution touchsurface. FIG. 39 shows the pickup contacts 3402 on the same layer as theresistive sheet 3404. Alternatively, as shown in FIG. 40, the contacts3502 can be placed in the interior rather than the edge of the resistivesheet 2504 using a two-layer approach, effectively allowing someelectrodes such as contact 3502 to be used for multiple patches 3506,3508.

FIG. 41 illustrates an example of dielectric-encoded passive positionsensing. A position of contact 3602, 3612 can be encoded to a singlepickup electrode by a dielectric code printed to the touch surface.Since the signal from the contact is capacitively transferred to theelectrode, it is possible to encode a dielectric pattern onto thesurface that modifies the signal as it is transferred to the pickupelectrodes. This dielectric pattern may be produced by etching, screenprinting, subtractive lithography, mechanical, or other means. Byknowing the dielectric pattern, it is possible to recover the positionfrom a single electrode by the results of de-convolution or otherinverse transforms 3610, 3614. Depending on the necessary contact areaand resolution, multiple such patches 3606, 3608 can be placed next toeach other to produce a complete touch surface, simplifying the code andincreasing the size of the code relative to the size of the patch ineach patch.

FIGS. 42 and 43 illustrate an example of continuous passive positionsensing using an array 3702 of non-linear elements 3704. The continuouspassive position sensing approach can be combined with row-columngrid-based position sensing to calculate the position of fingers. Due tothe non-linear response of the system to touch position, multipletouches on the same row or column can be distinguished. Therefore, itbecomes possible to use a row-column grid to calculate high-resolutionmulti-touch position. Instead of using a continuous resistive sheet, itis possible to replace the resistive sheet with a lattice of nonlinearreactive elements or a sheet material that has a nonlinear reactance.FIG. 42 shows a one-dimensional lattice for simplicity; similarprinciples apply to two-dimensional lattices. A signal injected intothis medium decomposes into a group of solitons (solitary excitations)that exhibit a distance- and frequency-dependent relative phase shift asthey travel through the medium. In FIG. 43, each line pattern showsincreasing distance from pickup. The soliton phase shifts can then beused to calculate the distance from each pickup point to the event,allowing determination of the event location. In one or moreembodiments, a nonlinear transmission line (lattice of nonlinearreactive elements) can be used with a multitude of pickup points. Insuch a case, the surface can be broken into zones or strips, with onearray covering each strip. The array also may be joined linearly, or ina matrix configuration with more than two connections to nearbyelements.

FIG. 44 illustrates an example of spatially-distributed coordinatedencoding. In one or more embodiments, the position of a contact ormotion event at the sensing surface can be determined by encodingcoordinates in physical variations of the surface which are then decodedfrom the signal generated by the event. An example of this is shown incross-section in FIG. 44: as a finger 3902 moves across a surface 3904with a varying height profile 3906, the detected signal 3908 reflectsthe profile variations along the direction of motion. Positioninformation can be encoded in these variations using a two-dimensionalself-clocking code, and subsequent signal processing by a coordinatedecoder 3910 can reconstruct the position and velocity of points alongthe trajectory 3912. This technique advantageously replaces an array ofelectrodes and associated electronics with a single electrode andamplifier, plus a textured surface to capture input motion, resulting inlow-cost gestural input surfaces.

FIG. 45 illustrates an example combination of TriboTouch with resistivetouch sensors. TriboTouch can be combined with additional sensingapproaches in order to use the existing physical designs, whileupgrading the capabilities of the system with the benefits thatTriboTouch technology offers, or to use the benefits of both approaches.Resistive sensors ordinarily use two layers 4002, 4004 coated with aresistive material, and separated by a small distance. There can beelectrodes 4006, 4008 along opposing edges of each layer, in verticaland horizontal directions, respectively. When the layers make contactdue to pressure from touch, a touch position is sensed. The electrodescan be used alternatively as receiver and as a voltage source todetermine the vertical and horizontal position of the touch. TriboTouchcan be combined with resistive sensors by placing pickups 4010 on thetop resistive sheet 4002 used in a resistive sensor. The pickups 4010can be used to derive the position of contacts on the top surface 4002.Note that since resistive sensors often use a full edge as a connector,additional or separable contacts may be needed. The resistive sensingcapability can be maintained by interleaving the processing of thesignals. Alternatively, in a quiescent state, the bottom layer 4004 canbe connected to a voltage source, while the top layer 4002 is used forTriboTouch. If a contact is of sufficient force to contact the bottomlayer, the TriboTouch system can detect the sudden large offset causedby contact with the bottom layer, hand off to the resistive system forresistive position detection, and begin interleaving at that time to useboth systems. Such an approach allows for reduced switching and reducedpower expenditure.

TriboTouch can also be combined with capacitive touch sensors. As shownin FIGS. 7 and 8, capacitive sensors operate by detecting the change ina transmitted electric field. In order to allow cooperation between thetwo systems, it is possible to connect a capacitive sensor ASIC directlyto the same pads as a TriboTouch system and achieve coexistence byinterleaved sensing. Since TriboTouch is capable of high-speedoperation, it is possible to use existing capacitive technology withoutsignificant change. Note that capacitive signals are of a known form andfrequency. Therefore, it is possible to operate non-transmittingelectrodes in TriboTouch mode while they concurrently receive the signalbeing transmitted by other electrodes. In such a case, filters may beused to reject the capacitive signals from the TriboTouch processingsystem, either using traditional frequency-domain filtering, or by usingsynchronous filtering in cooperation with the excitation signal producedby the capacitive sensor.

FIGS. 46 and 47 illustrate example combination of TriboTouch withinductive touch sensors. Inductive sensors operate by exciting an activestylus with a pulse of current using a matrix of wires. When a line isnot being used to provide excitation, it is possible to use these linesas TriboTouch receiver lines. Since TriboTouch does not transmit anysignals, the lines can be directly connected to the TriboTouch system.Note that if one end of the line is permanently attached to a fixedpotential rail 3902, the rail should be disconnected so that theTriboTouch signal can be read. This disconnection can be achievedthrough an electronic switch 3904. Alternatively, as shown in FIG. 47,if the inductive system is being operated with current pulses, theinductive system can be coupled capacitively, e.g., via capacitors 4202,4204, to the touch surface such that a continuous connection to a powerrail does not exist. An additional benefit of incorporating TriboTouchtechnology is the reduction in power use. Since inductive sensing usescurrent flow to form a magnetic field, it is power-hungry. By detectinginitial contact with the low-power TriboTouch technology, the inductivesensor can be disabled when there is no contact, leading to significantenergy savings when the system is quiescent.

In one or more embodiments, TriboTouch, TriboNoise, TriboNoiseTouch, orcombinations of those can be combined with other touch sensor types,such as surface acoustic wave, infrared, or acoustic touch sensors, aswell as with any of the resistive, capacitive, and inductive sensorsdescribed above. TriboTouch, TriboNoise, and TriboNoiseTouch can alsouse the electrode types described herein, except forspatially-distributed coordinate encoding electrodes, which can be usedwith TriboTouch and TriboNoiseTouch, as discussed above with referenceto FIG. 30.

Surface acoustic wave (SAW) touch sensors use transducers to produce anultrasonic wave that is absorbed when a finger makes contact. Thesurface is ordinarily glass or a similar hard material. This surface canbe patterned with a transparent conductive material to provide pickupsfor the TriboTouch system. No interleaving is necessary, since SAWsystems do not use electrical signals transiting the surface itself todetect position.

Infrared touch sensors produce infrared light that is absorbed when afinger makes contact. This surface can be patterned with a transparentconductive material to provide pickups for the TriboTouch system. Nointerleaving is necessary, since infrared systems do not use electricalsignals transiting the surface itself to detect position.

Acoustic touch sensors detect the specific sounds produced when anobject touches the sensed surface to detect position. This surface canbe patterned with a transparent conductive material to provide pickupsfor the TriboTouch system. No interleaving is necessary, since acousticsystems do not use electrical signals transiting the surface itself todetect position.

FIG. 48 illustrates an example computer system 4300. In particularembodiments, one or more computer systems 4300 perform one or more stepsof one or more methods described or illustrated herein. The processesand systems described herein, such as the processing system 312 of FIG.3, the noise processing system 1216 of FIG. 12 or the TriboNoiseTouchprocessing system 1916 of FIG. 19, can be implemented using one or morecomputer systems 4300. In particular embodiments, one or more computersystems 4300 provide functionality described or illustrated herein. Inparticular embodiments, software running on one or more computer systems4300 performs one or more steps of one or more methods described orillustrated herein or provides functionality described or illustratedherein. For example, the processing system 312 of FIG. 3, the noiseprocessing system 1216 of FIG. 12 or the TriboNoiseTouch processingsystem 1916 of FIG. 19 can be implemented as one or more methodsperformed by software running on the one or more computer systems 4300.Particular embodiments include one or more portions of one or morecomputer systems 4300. Herein, reference to a computer system mayencompass a computing device, and vice versa, where appropriate.Moreover, reference to a computer system may encompass one or morecomputer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems4300. This disclosure contemplates computer system 4300 taking anysuitable physical form. As example and not by way of limitation,computer system 4300 may be an embedded computer system, asystem-on-chip (SOC), a single-board computer system (SBC) (such as, forexample, a computer-on-module (COM) or system-on-module (SOM)), adesktop computer system, a laptop or notebook computer system, aninteractive kiosk, a mainframe, a mesh of computer systems, a mobiletelephone, a personal digital assistant (PDA), a server, a tabletcomputer system, or a combination of two or more of these. Whereappropriate, computer system 4300 may include one or more computersystems 4300; be unitary or distributed; span multiple locations; spanmultiple machines; span multiple data centers; or reside in a cloud,which may include one or more cloud components in one or more networks.Where appropriate, one or more computer systems 4300 may perform withoutsubstantial spatial or temporal limitation one or more steps of one ormore methods described or illustrated herein. As an example and not byway of limitation, one or more computer systems 4300 may perform in realtime or in batch mode one or more steps of one or more methods describedor illustrated herein. One or more computer systems 4300 may perform atdifferent times or at different locations one or more steps of one ormore methods described or illustrated herein, where appropriate.

In particular embodiments, computer system 4300 includes a processor4302, memory 4304, storage 4306, an input/output (I/O) interface 4308, acommunication interface 4310, and a bus 4312. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 4302 includes hardware forexecuting instructions, such as those making up a computer program. Asan example and not by way of limitation, to execute instructions,processor 4302 may retrieve (or fetch) the instructions from an internalregister, an internal cache, memory 4304, or storage 4306; decode andexecute them; and then write one or more results to an internalregister, an internal cache, memory 4304, or storage 4306. In particularembodiments, processor 4302 may include one or more internal caches fordata, instructions, or addresses. This disclosure contemplates processor4302 including any suitable number of any suitable internal caches,where appropriate. As an example and not by way of limitation, processor4302 may include one or more instruction caches, one or more datacaches, and one or more translation lookaside buffers (TLBs).Instructions in the instruction caches may be copies of instructions inmemory 4304 or storage 4306, and the instruction caches may speed upretrieval of those instructions by processor 4302. Data in the datacaches may be copies of data in memory 4304 or storage 4306 forinstructions executing at processor 4302 to operate on; the results ofprevious instructions executed at processor 4302 for access bysubsequent instructions executing at processor 4302 or for writing tomemory 4304 or storage 4306; or other suitable data. The data caches mayspeed up read or write operations by processor 4302. The TLBs may speedup virtual-address translation for processor 4302. In particularembodiments, processor 4302 may include one or more internal registersfor data, instructions, or addresses. This disclosure contemplatesprocessor 4302 including any suitable number of any suitable internalregisters, where appropriate. Where appropriate, processor 4302 mayinclude one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 4302. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 4304 includes main memory for storinginstructions for processor 4302 to execute or data for processor 4302 tooperate on. As an example and not by way of limitation, computer system4300 may load instructions from storage 4306 or another source (such as,for example, another computer system 4300) to memory 4304. Processor4302 may then load the instructions from memory 4304 to an internalregister or internal cache. To execute the instructions, processor 4302may retrieve the instructions from the internal register or internalcache and decode them. During or after execution of the instructions,processor 4302 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor4302 may then write one or more of those results to memory 4304. Inparticular embodiments, processor 4302 executes only instructions in oneor more internal registers or internal caches or in memory 4304 (asopposed to storage 4306 or elsewhere) and operates only on data in oneor more internal registers or internal caches or in memory 4304 (asopposed to storage 4306 or elsewhere). One or more memory buses (whichmay each include an address bus and a data bus) may couple processor4302 to memory 4304. Bus 4312 may include one or more memory buses, asdescribed below. In particular embodiments, one or more memorymanagement units (MMUs) reside between processor 4302 and memory 4304and facilitate accesses to memory 4304 requested by processor 4302. Inparticular embodiments, memory 4304 includes random access memory (RAM).This RAM may be volatile memory, where appropriate, and this RAM may bedynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Moreover,where appropriate, this RAM may be single-ported or multi-ported RAM.This disclosure contemplates any suitable RAM. Memory 4304 may includeone or more memories 4304, where appropriate. Although this disclosuredescribes and illustrates particular memory, this disclosurecontemplates any suitable memory.

In particular embodiments, storage 4306 includes mass storage for dataor instructions. As an example and not by way of limitation, storage4306 may include a hard disk drive (HDD), a floppy disk drive, flashmemory, an optical disc, a magneto-optical disc, magnetic tape, or aUniversal Serial Bus (USB) drive or a combination of two or more ofthese. Storage 4306 may include removable or non-removable (or fixed)media, where appropriate. Storage 4306 may be internal or external tocomputer system 4300, where appropriate. In particular embodiments,storage 4306 is non-volatile, solid-state memory. In particularembodiments, storage 4306 includes read-only memory (ROM). Whereappropriate, this ROM may be mask-programmed ROM, programmable ROM(PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM),electrically alterable ROM (EAROM), or flash memory or a combination oftwo or more of these. This disclosure contemplates mass storage 4306taking any suitable physical form. Storage 4306 may include one or morestorage control units facilitating communication between processor 4302and storage 4306, where appropriate. Where appropriate, storage 4306 mayinclude one or more storages 4306. Although this disclosure describesand illustrates particular storage, this disclosure contemplates anysuitable storage.

In particular embodiments, I/O interface 4308 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 4300 and one or more I/O devices. Computersystem 4300 may include one or more of these I/O devices, whereappropriate. One or more of these I/O devices may enable communicationbetween a person and computer system 4300. As an example and not by wayof limitation, an I/O device may include a keyboard, keypad, microphone,monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet,touch screen, trackball, video camera, another suitable I/O device or acombination of two or more of these. An I/O device may include one ormore sensors. This disclosure contemplates any suitable I/O devices andany suitable I/O interfaces 4308 for them. Where appropriate, I/Ointerface 4308 may include one or more device or software driversenabling processor 4302 to drive one or more of these I/O devices. I/Ointerface 4308 may include one or more I/O interfaces 4308, whereappropriate. Although this disclosure describes and illustrates aparticular I/O interface, this disclosure contemplates any suitable I/Ointerface.

In particular embodiments, communication interface 4310 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 4300 and one or more other computer systems 4300 or oneor more networks. As an example and not by way of limitation,communication interface 4310 may include a network interface controller(NIC) or network adapter for communicating with an Ethernet or otherwire-based network or a wireless NIC (WNIC) or wireless adapter forcommunicating with a wireless network, such as a WI-FI network. Thisdisclosure contemplates any suitable network and any suitablecommunication interface 4310 for it. As an example and not by way oflimitation, computer system 4300 may communicate with an ad hoc network,a personal area network (PAN), a local area network (LAN), a wide areanetwork (WAN), a metropolitan area network (MAN), body area network(BAN), or one or more portions of the Internet or a combination of twoor more of these. One or more portions of one or more of these networksmay be wired or wireless. As an example, computer system 4300 maycommunicate with a wireless PAN (WPAN) (such as, for example, aBLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephonenetwork (such as, for example, a Global System for Mobile Communications(GSM) network), or other suitable wireless network or a combination oftwo or more of these. Computer system 4300 may include any suitablecommunication interface 4310 for any of these networks, whereappropriate. Communication interface 4310 may include one or morecommunication interfaces 4310, where appropriate. Although thisdisclosure describes and illustrates a particular communicationinterface, this disclosure contemplates any suitable communicationinterface.

In particular embodiments, bus 4312 includes hardware, software, or bothcoupling components of computer system 4300 to each other. As an exampleand not by way of limitation, bus 4312 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 4312may include one or more buses 4312, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative.

What is claimed is:
 1. An apparatus comprising: an insulator coupled to one or more electrodes; wherein the one or more electrodes are configured to passively sense charge displacement or a change in characteristics of electromagnetic signals in an environment.
 2. The apparatus of claim 1, wherein the one or more electrodes are part of an array of electrodes.
 3. The apparatus of claim 1, wherein the insulator comprises a resistive sheet of known resistance per unit area.
 4. The apparatus of claim 3, wherein an input to the insulator sensed by each of the one or more electrodes is sensed as a function of the location of the input with respect to the insulator and the known resistance of the insulator.
 5. The apparatus of claim 1, wherein the insulator comprises an encoded dielectric surface.
 6. The apparatus of claim 1, wherein the insulator comprises one or more of the following: a lattice of nonlinear reactive elements; or a sheet having a nonlinear reactance.
 7. The apparatus of claim 1, wherein the insulator comprises a surface having known height variations.
 8. The apparatus of claim 1, wherein: the insulator comprises a resistive sheet of known resistance per unit area and pickup electrodes are placed on the resistive sheet in one or two dimensions; the insulator comprises an encoded dielectric surface; the insulator comprises a lattice of nonlinear reactive elements, or a sheet having a nonlinear reactance; or the insulator comprises a surface having known height variations.
 9. The apparatus of claim 1, wherein: the one or more electrodes include a top electrode and a bottom electrode and pickups are placed on the top electrode; a capacitive sensor system is coupled to the one or more electrodes; and an inductive sensor system is coupled to the one or more electrodes.
 10. An apparatus comprising: one or more electrodes configured to passively sense charge displacement or a change in characteristics of electromagnetic signals in an environment of the apparatus; an insulator coupled to the electrodes; and a computer-readable non-transitory storage medium embodying logic that is operable when executed to analyze the passively sensed charge displacement or change in characteristics of electromagnetic signals in the environment.
 11. The apparatus of claim 10, wherein the one or more electrodes are part of an array of electrodes.
 12. The apparatus of claim 10, wherein the insulator comprises a resistive sheet of known resistance per unit area.
 13. The apparatus of claim 12, wherein an input to the insulator sensed by each of the one or more electrodes is sensed as a function of the location of the input with respect to the insulator and the known resistance of the insulator.
 14. The apparatus of claim 10, wherein the insulator comprises an encoded dielectric surface.
 15. The apparatus of claim 10, wherein the insulator comprises one or more of the following: a lattice of nonlinear reactive elements; or a sheet having a nonlinear reactance.
 16. The apparatus of claim 10, wherein the insulator comprises a surface having known height variations.
 17. The apparatus of claim 10, wherein: the insulator comprises a resistive sheet of known resistance per unit area and pickup electrodes are placed on the resistive sheet in one or two dimensions; the insulator comprises an encoded dielectric surface; the insulator comprises a lattice of nonlinear reactive elements, or a sheet having a nonlinear reactance; or the insulator comprises a surface having known height variations.
 18. The apparatus of claim 10, wherein: the one or more electrodes include a top electrode and a bottom electrode and pickups are placed on the top electrode; a capacitive sensor system is coupled to the one or more electrodes; and an inductive sensor system is coupled to the one or more electrodes. 